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This paper describes a three-step method for the construction of codebooks meant for analyzing 
ePortfolio content. The first step produces a prototype based on qualitative analysis of very different 
ePortfolios from the same course. During the second step, the initial version of the codebook is 
tested on a larger sample and subsequently revised. Finally, during the third phase the codebook is 
applied to analyze ePortfolios’ contents and to chart trends of usage. We tested the codebook on the 
ePortfolios of 16 students attending a university blended course. This codebook and the method for 
building it enabled us to follow the ePortfolios’ evolution over the course, to observe students’ 
individual differences, to understand and guide students’ self-assessment, and to customize teachers’ 
and/or tutors’ interventions. Our method produces a tailored codebook for the examination of 
ePortfolio contents. 


The Need for the Analysis of ePortfolio Content 

The relevance of portfolios has been discussed 
extensively in the field of education, and portfolios 
have been used in “the broader areas of education and 
training, including work-based learning and the school” 
(Attwell, 2007, p. 40). The reference to an organized 
collection of documents produced by students is a 
recurrent, if not universal, feature of definitions of 
portfolio in the literature (Batson, 2002; Falls, 2001). 
Nevertheless, different types of portfolios are designed 
to reach diverse educational goals, such as supporting 
professional skills and documenting, evaluating, or 
presenting personal works. With the advent of new 
technologies, the portfolio evolved into ePortfolio , 
finding new areas of enrichment and novel contexts of 
implementation. Although there are some conflicting 
findings about the comparison of web-based and 
electronic portfolios’ usage (van Wesel & Prop, 2008), 
ePortfolios can transform portfolios from a thing to a 
process, to a content-management system for collecting, 
reflecting on, and sharing learning outcomes (Fitch, 
Reed, Peet, & Tolman, 2008). Much research 
conducted on the use of ePortfolios in education is 
aimed at analyzing how students perceive them 
(Bolliger, & Shepherd, 2010; Ritzhaupt, Singh, 
Seyferth, & Dedrick, 2008) or how they are used for 
assessment (Mason, Pegler, & Weller, 2004; 
Pelliccione & Dixon, 2008). 

Some research has also analyzed the process of 
interpreting and scoring ePortfolios by teachers (Schutz 
& Moss, 2004). Such research has shown how teachers 
create a “reasonable story” from the contents of 
ePortfolios and grade them according to such a story. 
However, the literature lacks a systematic identification 
of the emergence of core themes in ePortfolios. Indeed, 
the research by Schutz and Moss (2004) provided an 
insightful account of teachers’ interpretation of 
ePortfolios’ contents, but does not investigate directly 
the actual contents of the ePortfolios, which is a 


complementary methodology for understanding what 
the story contained in each ePortfolio is about. In 
particular, readers’ strategies may not guarantee an 
awareness of the least noted themes within the 
ePortfolios. 

The literature on assessment also provides good 
evidence for the value of rubrics (i.e., matrices 
containing assessment criteria and benchmarks of 
performance) in diverse settings (Hafner & Hafner, 
2003; Lasater, 2007; Roblyer & Wiencke, 2003; 
Saddler & Andrade, 2004). Rubrics may also be used 
for evaluating students (and educational activities) 
through a rubric-based ePortfolio assessment, 
especially considering self-regulation, critical skills, 
and active participation. However, the use of rubrics is 
not exempt from issues related to the validity of 
assessment (Moskal & Leydens, 2000). Mabry (1999) 
claimed that rubrics may raise validity problems 
similar to those raised by test-based assessment. The 
author argued convincingly that rubric-based 
assessment generally prescribes what counts as 
satisfactory performance before a performance is 
realized, even when it is not easy to predict what the 
students would execute (Mabry, 1999). Therefore, any 
performance that differs from the predicted standard is 
discouraged (Mabry, 1999). One possible (partial) 
solution to this problem is to monitor periodically the 
range of contents/performances collected in the 
ePortfolios and to tune the rubric criteria and 
benchmarks to the observed context. In this article, we 
describe the construction of a codebook as a useful 
tool for monitoring the core themes of the ePortfolios 
created by students. Particular attention is paid to the 
students’ metacognitive process; this is a crucial 
aspect in the implementation of an ePortfolio as a tool 
for reflection about the learning process. Indeed, using 
ePortfolios for reflection is a very effective strategy in 
education (Kabicher, Kriglstein, Figl, & Motschnig- 
Pitrik, 2008). Even though the resulting final 
codebook that we present in this paper is tailored to 
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the specific course we observed, the method we 
propose to construct can be considered as a model for 
developing a customized codebook in any context 
where portfolios is considered as an important part of 
the learning experience. 

The ePortfolio and its Contents 

Through the analysis of ePortfolios, it is possible to 
track the progress and evolution of the learning 
processes (Barrett, 2001). We consider ePortfolios to be 
organized collections of artifacts, produced either 
individually or collectively using various formats (e.g., 
video, graphics, or text). In students’ hands, ePortfolios 
can be reflexive tools for self-assessment, self¬ 
regulation, critical skills, and active participation 
(Jenson, 2011). We agree that ePortfolios are valuable 
tools for making students “active in formative 
assessment rather than passive receivers of graded 
results” (Pelliccione & Dixon, 2008, p. 752). In this 
sense, ePortfolios may encourage assessment for 
learning rather than assessment of learning (Stiggins, 
2002 ). 

The use of ePortfolios promotes so-called “folio 
thinking,” a term coined by Helen Chen (2004) to 
indicate the mental habit of building connections 
among experiences, skills, and artifacts and of making 
these connections visible to readers, but especially to 
the students authoring the ePortfolios. Students occupy 
a central position because by creating ePortfolios they 
are actually encouraged to take responsibility for their 
own learning (Paulson, Paulson, & Meyer, 1991). The 
personal and informal communication that may be 
embedded in the ePortfolios can support motivation and 
can act as further leverage for learning. At the same 
time, the teacher can monitor, direct, and guide the 
learning process, since the ePortfolio also gives 
information about the areas to be improved. 

ePortfolios can be structured around three distinct, 
yet interrelated themes: the first one is dedicated to 
reflection, the second one to documentation, and the 
third one to collaboration/mentoring (Zubizarreta, 
2004). Accordingly, the contents of ePortfolios may 
regard: (1) the philosophy of learning or narrative 
reflection upon the processes in progress, (2) the 
products of learning (e.g., course descriptions, 
curriculum, tutoring), (3) the evidence of learning (e.g., 
research articles, critical essays), (4) the assessment of 
learning (e.g., feedback, scores from tests), (5) the 
importance of learning (e.g., practical applications, 
personal growth, emotional value of learning), (6) 
learning objectives (e.g., improvement plans, goals), 
and (7) appendices (e.g., selected documentation of 
didactic materials). However, research seems to focus 
on the use of ePortfolios for assessment without dealing 
directly with their contents, so that few empirical 


studies investigate systematically the contents of 
ePortfolios. For example, Chang, Tseng, Chou, and 
Chen (2011) examined the reliability and validity of 
peer assessment for web-based portfolios, discussing 
the limits of peer assessment and the need to develop 
peer assessment skills. Mason et al. (2004) discussed 
the use of ePortfolios for assessment tools application. 
Considering that ePortfolios and learning objects 
involve the same fundamental technology and rely on 
the same capabilities for selection and re-use, 
ePortfolios are here proposed as the final assessment of 
a course designed around learning objects. Buzzetto- 
More (2010) tested the efficacy of ePortfolios and 
investigated students’ perceptions of ePortfolios as a 
tool for enhancing the understanding of learning goals 
and reflection on their own knowledge and skills. 

Although these studies are interesting, none of 
them deals directly with the contents of ePortfolios. The 
few studies extant that analyze the content of portfolios 
are designed to gain “insight into students’ rhetorical 
approaches to portfolio composition; their decisions 
related to selection of content, and the organization and 
design of their portfolio” (D’Angelo, 2009, p. 1) or to 
obtain feedback about how students use ePortfolios 
(Kabicher et al., 2008). The specificity of our 
contribution is to identify and classify the issues 
emerging from students’ ePortfolios while they are 
under development, considering both the temporal 
dimension and students’ individual differences. We 
describe, therefore, the process of building a codebook 
that may be used to identify themes emerging from 
content analysis of ePortfolios. Such analysis will 
provide an overview of students’ reflections contained 
in their ePortfolios. 

A Blended University Course: The 
Context of this Study 

The course analyzed in this paper was delivered in 
a blended learning mode (BL), in which computer- 
mediated learning and teaching in presence were 
integrated and combined (Bersin, 2004). In addition, the 
course used different teaching methods, diverse modes 
of study (e.g., individual, dyads, small group, and 
plenary activities), and a variety of tasks and artifacts 
(e.g., see Ligorio & Cucchiara, 2011; Ligorio, 
Loperfido, Sansone, & Spadaro, 2010). 

The course was divided into five modules covering 
the following contents: online educational models, 
learning objects, online identity, new trends, and a final 
module dedicated to the collaborative construction of a 
grid of indicators meant to analyze online courses. Each 
of the modules, lasting a week, was introduced by the 
teacher’s lecture, followed by a discussion via web- 
forum that was coordinated by an e-tutor. The lectures 
were usually scheduled as follows: (a) discussion of 
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the topic covered during the previous module (during 
the first meeting the teacher presented an overview of 
the organization of the course), (b) introduction of a 
new topic for the next module, (c) discussion of the 
progress of on-line activities (during the first meeting 
the teacher introduced the functions of the e-learning 
platform), and (d) assignments for the following 
module. 

The platform used, Synergeia 

(http://bscl.fit.fraunhofer.de), was designed to 
support online collaborative learning (e.g., Ligorio & 
Veermans, 2005). This platform allows both 
synchronous (chat) and asynchronous (web forum) 
communication and contains tools for the 
construction of concept maps (i.e., Map Tool), a 
shared calendar, and spaces for uploading and 
sharing files. In Synergeia, each module was 
represented by a folder containing the readings 
selected by the teacher (e.g., digital documents, 
slides, links to websites), and several areas for 
discussion via web-forums where students could 
discuss the materials and topics in the modules. 

Considering the relevance of working in groups for 
obtaining collaborative learning (Dillenbourg, 1999), 
the participants were divided into two groups that were 
formed randomly, each consisting of eight students. 
The groups were asked first to discuss online the 
educational materials and then to build collaboratively a 
concept map that summarized the contents of the 
module and a document describing their collaborative 
learning process. Each student was required to be active 
in the group and to take responsibility for achieving 
common goals, interpreting a role designed in reference 
to the tasks (e.g., leader responsible for the cognitive 
map, tutor of the group discussion) that had been 
assigned by the teacher. At the third module, in order to 
promote socialization among all participants, the groups 
were re-combined, and two new groups, again 
consisting of eight students each, were formed. 

The Structure of the ePortfolio 

Throughout the course, students were required to 
create and manage a personal ePortfolio that adhered to 
the following structure and contained: 

• A folder named The Best of Me in which, at 
the end of each module, students uploaded a 
selection of the artifacts produced throughout 
the module, either individually or 
collaboratively. Such an artifact might be a 
post in a discussion that the student considered 
to be particularly relevant, a written review of 
the material read, a contribution to a map, or 
other significant elements that represented the 
best of their participation in the module. This 


was a limited selection of not more than four 
artifacts per module. In addition, each artifact 
had to be accompanied by a comment that 
explained why it had been selected. The Best 
of Me folder was the core of our ePortfolios. 

• A folder titled Personal Space, through which 
students could present themselves to their 
teachers, tutors, and peers through links, 
images, video, and text (e.g., self-descriptions, 
expectations, free thoughts, links to personal 
blogs or Facebook profiles). Students could 
expand and enrich this space as they liked 
throughout the entire course. 

• A self-evaluation form, to be updated at the end 
of each module, with information about the role 
the student served during the module (e.g., 
responsible for the map, tutor of the group 
discussion) and his or her self-assessment 
regarding the individual and collaborative tasks 
completed (see Appendix A); 

• A folder called Balance of the Modules 
containing one web-forum for each module 
covered during the course. These web-forums 
contain discussion statements, impressions, 
and ideas about the various activities and 
about the ePortfolio itself. The Balance of the 
First Module asked about how students 
approached the course. In the second module, 
students were asked to reflect on their self- 
assessment—in particular, on the relationship 
between participation/learning, about the role 
they played during the course, and about 
collaboration within the group during the first 
module. In the third module, the task was to 
reflect on specific activities the students had 
performed, such as discussions around the 
readings, the construction of the cognitive 
map, and the description of the process of 
collaboration. In the fourth module, the 
assignment asked students to reflect on the 
role-play and on the re-composition of 
groups. The balance of the fifth module was 
included in the forum for the final discussion, 
described below. 

• Finally, a forum called Big Balance featured a 
global and final reflection on the course. Here 
the task was to discuss freely the Course of 
Psychology of E-Learning, and each student 
was asked to describe her or his final 
impressions. 

To build the codebook, we analyzed the notes posted in 
all of these sections. Each student entered a different 
number of notes; some students, furthermore, carefully 
developed all of the sections, while others left some 
sections empty. 
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Objectives 

The objective of this paper is to describe the 
development of a procedure for building a tailored 
codebook to analyze ePortfolios built in an e-learning 
or blended course. In general, codebooks can be used to 
enhance teachers’ awareness of the issues reflected on 
by students and to obtain feedback on students’ 
experiences and perceptions. Codebooks can also 
constitute a working tool for the 
development/improvement of a rubric for the 
ePortfolios’ assessment, while overcoming the validity 
issues that may be associated with the rubric’s usage, as 
discussed by Mabry (1999). Through the codebook, it is 
possible to obtain an overview of the range of themes 
that students actually select; this allows the codebook to 
become a tool able to guide the construction of the 
rubric. Finally, codebooks are good tools for research 
on ePortfolios, facilitating analysis of the contents that 
students include in them. In this paper, we refer mainly 
to this latter option by describing the creation, 
development, and use of codebooks as tools in the 
researcher’s hands. We consider the method used to 
create a codebook as the main outcome of this study. 
The codebook should be regarded as a tool for inquiry 
about ePortfolios’ content and, more specifically, for: 

• analyzing the distribution of themes and 
categories across the different sections of 
ePortfolios, 

• observing the evolution of the themes over 
time in order to have a diachronic vision of the 
ePortfolios, 

• facilitating the review and analysis of 
ePortfolios in following iterations of the same 
course, and 

• triggering the construction of codebooks in 
similar e-leaming and/or blended courses. 

The content of ePortfolios built during the university 
course described above is the object of our analysis. 

Method 

The method we propose is based on an inductive 
approach inspired by grounded theory and content 
analysis. We consider such approaches useful for an 
exploratory analysis of the type of data we collected. 
Glaser and Strauss (1967) proposed grounded theory as 
a set of procedures for the inductive development of 
theoretical propositions of an increasing level of 
abstraction, starting from the analysis of data. In this 
framework, theory is developed from the data through 
an iterative process of defining, modifying, and 
redefining the categories of analysis of the empirical 
data (Glaser & Strauss, 1967). The term grounded 


emphasizes the idea of a theory generated through an 
interactive process, in which the theory is developed 
from data (Glaser & Strauss, 1967). Data analysis is 
carried out through a coding process that seeks to find 
the conceptual category that best expresses the meaning 
of a piece of data (Glaser & Strauss, 1967). 
Systematically comparing the different conceptual 
categories, one is able to abstract a more general 
meaning. This process should not be done linearly, but 
circularly (Glaser & Strauss, 1967). The coding of the 
data leads to the formulation of new hypotheses that 
may differ from the initial ones; the circularity is 
considered to be a strong point of the grounded theory 
approach, as it forces the researcher into a continuous 
process of interpretation and reflection on every step 
(Glaser & Strauss, 1967). 

Content analysis is a methodology for the 
objective, systematic, and quantitative analysis of the 
content of communication (Ghiglione & Blanchet, 
1991; Hsieh & Shannon, 2005). This research method is 
based on the subjective interpretation of the content of 
text data through a systematic classification process of 
coding and identifying themes or patterns (Hsieh & 
Shannon, 2005). A widely adopted operating procedure 
for content analysis breaks down communicative units 
into simple elements (called units of classification) that 
are then categorized (Hsieh & Shannon, 2005). The 
choice of categories is crucial and difficult, since 
meanings are directly dependent on the context, while 
coding is de-contextualizing and is implemented 
through a recording of data to obtain a codebook (Hsieh 
& Shannon, 2005). A codebook usually consists of 
categories of analysis that can be established a priori 
based on theoretical references, or a posteriori when 
extrapolated from the data using a grounded theory 
approach, as in our case. Specifically, we went through 
the process of coding, which means that we searched 
for a word or a short sentence to which it was possible 
to assign a summative, salient, essence-capturing, 
and/or evocative attribute (Saldana, 2009). By 
systematically comparing the different conceptual 
codes that had been assigned, it was possible to abstract 
from the data more general categories or themes. This 
process was done through a circular route. 

Participants 

In this section, we describe briefly the students 
participating in our study. The participants were 16 
students (12 female, four male) attending a specialist 
course for future Work and Organizational 
Psychologists. The mean student age was 25. The 
course was held at the University of Bari, and it was 
labeled Psychology of E-Learning. The students all 
came from the region where the city of Bari is located, 
in the South of Italy. This is a public university; the 
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students therefore came mostly from the middle 
social-class. The faculty of Psychology at the 
University of Bari has a good reputation, and the 
program generally attracts motivated students who are 
willing to invest time and energy in their education. In 
this course, the students already knew each other 
because they had attended the first segment of the 
university path (three years) that was mandated before 
they could enter the specialized level. To enroll in the 
class, they had to have passed an admissions test. The 
blended mode for delivering the course was optional, 
offered as an alternative to the traditional mode. Those 
students that, for any reason, did not want to join the 
course in the blended mode could attend lectures and 
have a final colloquium with the teachers without 
doing any activity online. In fact, this is the usual way 
of passing courses in Italy. 

Description of the Process: Analyzing ePortfolios 

The process of creating the codebook was divided 
into three steps: (a) an exploratory phase for obtaining a 
rough version of the codebook, (b) a phase during 
which the initial version of the codebook was tested on 
a larger sample of students from the same course and 
modified according to the results, and (c) the final 
phase in which the codebook was used to analyze the 
ePortfolios’ contents and to chart trends in their usage. 

First step: Building the prototype of the 
codebook. The first step was designing an outline of 
the desired codebook by qualitatively analyzing a small 
sample of ePortfolios. To accomplish this first phase, 
three different actions were performed. First of all, we 
selected and analyzed the two most diverse ePortfolios 
of the course. These ePortfolios were selected 
according to the following criteria: 

• gender difference (male and female); 

• different levels of computer skills and 
competence declared by the students at the 
beginning of the course in the self-assessment 
questionnaire, a five-point Likert scale ranging 
from 1 ( low competence, defined as infrequent 
use of the computer in everyday life) to 5 
0 high competence, defined as very frequent use 
of the computer in everyday life); and, 

• different levels of participation in the course, 
calculated from the number of notes posted in 
each ePortfolio. 

Two students were selected as cases for testing our 
method. To protect their anonymity, we will call them 
Max and Lara. They posted a total of 20 notes: Max 
authored eight sets of notes, and Lara authored 12. 

The second action consisted of segmenting the 
notes into units for analysis. Each unit corresponded to 


phrases having a recognizable meaning that was 
different from the preceding and from following 
phrases. 

The third action was meant to define themes and 
categories. To accomplish this aim, we used a qualitative 
approach inspired by grounded theory and content 
analysis. This method allowed us to identify five main 
themes that emerged from the first two selected 
ePortfolios: Technology, Participation, Competences, 
Assessment, and Self. Each theme was composed of at 
least one, and up to three, categories. For example, the 
theme Self was composed of two categories: individual 
characteristics, or references made to personal 
characteristics and their implications for the activities 
performed during the course; and emotional aspects, 
referring to moods and expectations. The categories 
described how the theme was actually perceived by the 
students. Appendix B describes in detail the themes and 
categories that emerged from this first step. 

A small percentage of sentences (5%) were 
excluded from the coding process because they were 
considered to be irrelevant or ambiguous. Therefore, 36 
total segments were used as the corpus of data for this 
first phase of the codebook construction. Two 
researchers, after sharing the objectives of the research, 
assigned codes to the corpus of data and independently 
developed an initial set of themes. Afterwards, they 
discussed the codes and themes to obtain a shared 
coding scheme. The controversial cases (about 18%) 
were also coded by a third researcher and discussed 
until 100% agreement was reached. 

Second step: Testing and modifying the first 
version. After identifying themes and categories, we 
tested the version of the codebook obtained at the end 
of the first phase on the remaining 14 ePortfolios 
produced during the course. The corpus of data used in 
this phase consisted of 117 notes, divided into 353 
segments. In this phase, the analysis was performed by 
the same two judges who initially had worked 
independently. In this first step they reached an 
agreement of 75.71%. After discussing the 
controversial cases with a third researcher, an 
agreement of 98.3% was obtained. The few notes on 
which agreement was not reached were erased from the 
data set, since the researchers agreed that they were not 
relevant for this analysis. During this step, a few 
categories were revised as follows: 

• The category Modality of Work was redefined 
to include considerations about the success or 
the problematic aspects of the entire course. 

• The category Emotional Aspects was extended 
to incorporate references to expectations for 
the future. 

• A new category called Phatic was introduced, 
which we then included in the participation 
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theme. This category refers to expressions 
intended to open a dialogue with other 
students or the teacher by asking, for instance, 
for feedback or inviting others to express 
opinions (e.g., “What do you think?”). 

• The theme of Sociality was extended to 
include aspects of pro-sociality, which is 
defined as an attempt to meet the needs of 
others (e.g., “this allows the others to better 
understand our point of view”). 

Appendix C shows the codebook in its final version, 
with the five themes and all the categories, 
accompanied by examples. 

Third step: Application of the codebook. Finally, 
we used this version of the codebook to investigate the 
structure of the ePortfolio and its contents, the 
distribution of the themes across the different sections, 
and the evolution of the contents over time, in order to 
achieve a diachronic understanding of the students’ 
self-evaluation. In the following paragraphs, we recount 
briefly our findings. 

Results 

The most frequent theme we found in our data was 
Assessment, which comprised 36% of the total 
frequency, confirming that ePortfolios play a role 
connected to self-evaluation and self-assessment. 
Participation (29%) was the second most frequent 
theme, followed by Self (16%), Competence (11%), 
and finally Technologies (8%). We expected this latter 
theme to be more dominant in students’ reflections, 
because of both the contents of the course and the 
required online activities, but surprisingly it was 
discussed only briefly at the beginning of the course. 
Probably students quickly became used to technology, 
so that it became an invisible part of the course. Other 
research proves that there is no significant difference 
between online and offline portfolios (Lunt, 2009). It is 
the activity of maintaining a portfolio, regardless of its 
format, and the formative feedback received that permit 
students to improve their performance in terms of self- 
assessment and reflection. 

According to Figure 1, Self-Assessment appeared 
with the greatest frequency (18%). This result is not 
surprising, considering that this category contains 
student reflections, which probably were perceived to 
be the core aspect of the ePortfolio. 

Figure 1 also shows that the second most frequent 
category was Individual Activities (13%), which 
remains an important aspect of the course, whereas 
“Group activities” had a relatively low frequency (4%). 
However, we noticed that often the contents categorized 
as Individual Activities referred to activities that were 
meant to support the groups. For instance, students 


often talked about role-taking, which was technically an 
individual activity but in fact was intended to support 
group activities, discussions, and the collaborative 
construction of products (Brown & Campione, 1990; 
Hare, 1994; Slavin, 1999). In general, students 
considered the role-taking to be very relevant, acting as 
a hinge between the individuals and the group. In the 
light of this result, during the planning of the following 
edition of the course, the teacher valued role-playing as 
a scaffold to improve students’ participation. 

The category Modality of Work had a frequency of 
12%. The Modality of Work category referred in 
particular to the introduction of the blended mode of 
course delivery, which implied for these students a set 
of novelties, such as different time management, the 
alternation between online and offline contexts, and the 
need to develop new learning strategies. We found this 
result very interesting for teachers, who might want to 
invite students to discuss explicitly their learning 
strategies. 

The Emotional Aspects (9%) appeared to work as a 
glue linking personal expectations, pressure for 
performance, and personal interests. The remaining 
categories showed a fairly low rate of less than 7%. 

The Themes in Different Sections 

The various sections of the ePortfolio (e.g., 
personal space, web forum, Big Balance, self- 
evaluation form) differed from one another in terms of 
requests and aims; therefore, we expected to find 
among them different distributions of core themes. 
Indeed, some themes proved to be central in some 
sections and rare in others (see Figure 2). 

This analysis allowed us to see what functions 
each section covered within the ePortfolios. Our 
results show that: 

• In the web-forum, notes about Participation 
(31%) and Assessment (27%) prevailed. 

• In the Big Balance used at the end of the 
course, the most frequent themes were the 
same as those in the web-forum, but in an 
inverse ratio: the most prevalent theme was 
Assessment (37%), followed by Participation 
(26%). The theme Competence reached here 
its highest percentage in comparison to other 
sections (19%), probably as a general 
recognition of the competencies acquired 
thorough the entire course. 

• The self-evaluation form, completed at the end 
of each module, contained a large percentage 
of Assessment (52%) and contained rather 
numerous references to Participation (33%). 
Both social and individual learning were 
recognized by the students as important 
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Figure 2 

Distribution of the Themes Through the Sections ofePortfolios 
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aspects of this learning experience. 
Participation and assessment were also 
perceived by students as connected aspects of 
the self-evaluation. On the contrary, the 
dimension Self reached only 5% of the 
frequency, indicating that it was not so 
obvious for these students to express personal 
issues or emotional experiences. 

• The personal space was focused on 
Assessment (47%), while Participation seemed 
to be less relevant (17%); here the focus was 
on self-reflection. This result was unexpected, 
given that the personal space was designed to 
focus more on self-presentation and sharing of 
emotions. 

Evolution of Themes Through the Course 

One of the characteristics of an ePortfolio is its 
temporal evolution. In this course, the flow of time was 
marked by the portfolio’s modular structure. In order to 
investigate the variation over time, the forums called 
Statement and Evaluation and the self-evaluation forms, 
both active at the end of each module, were compared. 
The Big Balance, which was aggregated to the fourth 
module, coincidentally was completed at the course’s 
conclusion. The personal space was not considered, as 
it was a personal activity and was not structured in 
modules. Figure 3 shows the distribution of the themes 
in relation to time, comparing the sections completed at 
the end of each module. 


Figure 3 shows that the theme Technology had a 
higher frequency at the beginning of the course (18%) 
but became less relevant in subsequent modules. Our 
interpretation of this trend is that students reflected on 
the use of technology at the beginning of the process of 
instrumental genesis, which required students to 
appropriate technological tools and integrate them into 
their practices (Ritella & Hakkarainen, 2012); however, 
afterwards technology generally became an invisible 
background for other activities (Engestrom, 1987). It is 
interesting to note that in modules 1 and 2, Evaluation 
exceeded 40% of frequency but dropped to 30%-35% in 
subsequent modules. In contrast, Participation was low 
in the first module (16%), but reached 34% in the 
second module and remained between 26% and 33% in 
subsequent modules, probably because, after the initial 
modules, students introduced the narrative of their 
learning and their participation as part of their 
reflection. This aspect seems in line with the folio 
thinking idea (Chen, 2004), suggesting that the 
ePortfolio triggered the construction of a narrative in 
which experiences (in terms of participation) and skills 
(of which students became aware through the 
evaluation) were linked. 

The theme of Competence reached its peak in the 
third module (21%), as did the theme of the Self (19%). It 
is worthwhile to recall that for the third module, groups 
were remixed; this probably gave students the opportunity 
to strengthen in a new group the competencies acquired in 
the previous group, and, consequently, their attention to 
their competencies was reinforced. 


Figure 3 

Frequencies of Theme in Each Module 


50 



IModule 2Module 3Module 4Module 


■ Technologies 

■ Participation 

■ Competence 

■ Evaluation 

■ Self 































Impedovo, Ritella, and Ligorio 


Developing Codebooks 169 


The analysis of the temporal evolution of 
ePortfolios allows the researcher, as well as the teacher 
and/or tutor, to understand the trajectory along which 
an ePortfolio is evolving. This information can be used 
to regulate the activity of maintaining the ePortfolio, as 
well as for understanding where and when there is a 
need for intervention or adjustments. 

Individual Style 

The codebook can be used to sort out the themes 
used by each student and, consequently, to reflect about 
individual learning paths. For example, considering the 
two initial ePortfolios examined, the majority of notes 
from Max (who was accustomed to using technology) 
fell into the emotional aspects category (42%), with 
references to moods and expectations. Lara (unfamiliar 
with the use of technology) had a greater focus on 
content (24%), with many references to the educational 
materials she read during the course. Future research 
might use the codebook to investigate more thoroughly 
the personal style that emerges from students’ 
ePortfolios. 

Discussion 

The three-step process of building a codebook that 
we developed has been applied to analyzing the core 
themes referred to by students when they completed the 
various sections of an ePortfolio in a blended university 


course. In particular, we analyzed differences across the 
ePortfolios’ sections, both throughout the course and in 
individual students’ ePortfolios. Such analysis permits 
monitoring the sections of the ePortfolios at different 
moments of the course. The process of building a 
codebook is summarized in Table 1, along with some 
suggestions about how it can be used in other courses. 

As indicated in Table 1, the application of the 
codebook to other contexts must be tailored to fit 
different curricula and assessment goals. Furthermore, 
it is useful to consider that the codebook, as we used it, 
did not contain all possible aspects analyzable in 
ePortfolios. In our case, some aspects were neglected 
because they were not relevant within the structure of 
the course (e.g., level of team interaction, relevance to 
learning objectives, media effectiveness). In particular, 
the reflection in our ePortfolios remained mostly 
individual, whereas the assessment of group and 
collective activity was implemented in other parts of the 
course, in connection with the group activities. 

In general, we can conclude that although we 
recognize that the literature does not clearly point to the 
need for a codebook, we believe this tool enables a full 
examination of ePortfolio contents, since any aspect 
considered when planning and developing the 
ePortfolios can be taken into account. Such a tool can 
be considered as a new lens for understanding what 
content students use when creating and maintaining 
ePortfolios and how this content changes over time and 
across the ePortfolios’ sections. 


Table 1 

The Process of Building a Personalized Codebook: Three Steps 


_ Steps _ 

1: Building a prototype 


2: Testing the initial 
version of the codebook 
on a larger sample and 
modifying it according to 
the results 


3: Using the codebook to 
analyze ePortfolios’ 
contents 


_ Aims _ 

Creating a first draft of 
the codebook 


Verifying the solidity of 
the categories and 
finalizing the codebook 


Using the codebook to 
assess the content of the 
various sections of the 
ePortfolios and its 
evolution 


_ Application _ 

Selecting and analyzing 
two very different 
ePortfolios 

Analyzing all the 
students’ ePortfolios, 
changing labels and the 
content of categories and 
adding new categories 
when needed 

Analyzing the distribution 
of the frequencies at 
different times and into 
the various spaces of 
ePortfolios, analyzing 
individual codebooks 


How it can be applied in 

_ other courses _ 

With a larger sample, 
more than two different 
ePortfolios could be 
selected 

Adapting the categories to 
the aims of the course and 
to the activities 
composing the ePortfolios 
(e.g., task, skills, level of 
group versus individual 
performances, creativity). 
Statistical analysis of all the 
ePortfolios obtained, 
comparing different 
sections, different times, 
individual performances, 
group performances, media 
effectiveness, and etc._ 
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Although the codebook has many advantages, 
applying it in a course is a rather difficult and long 
process; involvement by at least two experts is 
advisable. By experts we mean researchers, tutors, or 
even teachers who have a clear understanding of the 
structure of the ePortfolio and its general function 
within the course. Furthermore, knowledge about 
content analysis and grounded theory is required. Time 
is needed to establish the themes and categories, to 
cover the three steps we have outlined, and to discuss 
and negotiate the results. 

Nevertheless, we are convinced that this tool can 
support a good understanding of how students perceive 
and use the ePortfolios. Additional, more extensive 
applications of the codebook (e.g., looking at personal 
profiles) may guide students’ self-assessment as part of 
the learning process and customize teachers’ and tutors’ 
intervention in an appropriate manner. At the same 
time, considering the set of ePortfolios produced in a 
course, a story told from different personal observation 
points about learning experiences in the course could 
emerge. Therefore, the codebook could also be used to 
define specific profiles of participation and to observe 
differences and similarities among members of the 
same group. Further instantiations of the codebook in 
other courses and contexts could improve its power and 
make it more solid and reliable. 
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Appendix A 

The Self-Evaluation Form 


Student First and Last Name:_ 

Instructions: This form should be updated at the end of each module. Please fill in the column corresponding to the 
module just completed by assessing yourself about the statement reported on the left along a scale from 0 to 4 (0 = 
not at all, 1 = slightly, 2 = somewhat, 3 = moderately 4 = extremely). 



MODULE 1: 
Learning and 
Technology 

MODLE 2: 

Learning Object, 
Open source 

MODULE 3: 
Digital Identity 

MODULE 4: 
New trends 

Role-play (Self-evaluation of the role played. Do not answer if you did not play any role.) 

How much do you think the role 
supported your participation in the 
activities? 





How much do you think the role helped 
you in learning the contents of the 
module? 





How much do you think the role helped 
you acquire learning skills? 





Writing Reviews 

How much do you think writing a 
review on the reading material helped 
the knowledge building of your 
group? 





How much do you think you learned 
from writing the review? 





How much do you think writing the 
review helped you acquire learning 
skills? 





Web-forum discussion 

How much do you think you 
participated in the discussion? 





How much do you think the discussion 
helped you in learning the content of the 
module? 





How much do you think the 
participation in the discussion helped 
you acquire learning skills? 





Classroom activities 

How much do you think classroom 
activities helped you in learning the 
content of the module? 





How much do you think classroom 
activities helped you acquire learning 
skills? 





Building cognitive maps 

How much do you think you contributed 
in building the cognitive map? 





How much do you think participating in 
the construction of the map helped you 
acquire learning skills? 
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Group activities 

How much did you feel part of your 
group? 





How much did you feel part of the 
larger group? 





Self-evaluation 

How well do you think you are capable 
of assessing yourself? 





List the contribution you consider the 
most important for the module (a note of 
a discussion, a product, etc.)- 
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Appendix B 

First Themes and Categories 


Theme 

Description 

Categories 

1. Technology 

Reference to difficulties regarding the online environment and 
reflection on their technological skills 

Relationship with 
technology 

Tools 

2. Participation 

How students reflect on the modality of participation and 
interaction, and how they perceive group dynamics 

Sociality 

Individual activities 

Group activities 

3. Competence 

How students reflect on their skills 

Individual competence 
Group competence 

4. Assessment 

How students evaluate themselves and the activities of the course 
(metacognition) 

Content 

Self-assessment 

Modality of work 

5. Self 

Personal and emotional aspects emerging from participation 

Individual features 
Emotional aspects 
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Appendix C 
The Final Codebook 


Themes 

Categories 

Description 

Examples 

1. Technologies 

l.a. Relationship with 
technology 

Reference to the relationship with the “I never had any talent for 
technology technology.” 


l.b. Tools 

Reference to a specific tool 

“I created a wiki note for 
discussion.” 

2. Participation 

2.a. Sociality 

Reference to the social aspects of 
participation such as sharing, mutual 
exchange, pro-social attitudes 

“I felt encouraged to 
participate.” 


2.b. Individual 
activities 

References to individual activities. 

“To me, it was very 
interesting to play the role of 
the synthesizer.” 


2.c. Group activities 

References to the composition, 
organization and group dynamics 

“The identity of the group 
was strong.” 


2.d. Phatic 

Supporting discussion and asking for 
feedback 

“Well, what about it?” 

3. Competence 

3.a. Individual 
competence 

References to personal skills 

“This role develops the 
ability to extract the key 
elements of a module.” 


3.b. Group competence 

References to skills gained while 
working in a group or used to carry 
out group work 

“We developed analytic 
skills.” 

4. Assessment 

4.a. Content 

References to the contents of the 
educational materials 

“It was very interesting to 
understand what really is the 
aim of e-learning.” 


4.b. Self-assessment 

Critical reflections on their own work “Now that my ePortfolio is 

completed, I realize the 
progress I’ve made.” 


4.c. Modality of work 

References to the method used to 
perform an activity, even concerning 
the problematic aspects 

“I have the impression that 
this discussion has been too 
short for the type of 
argument.” 

5. Self 

5.a. Individual 

References to personal characteristics “I love to be able to draw 


characteristics 

and to the implications for the 
activities 

conclusions because, in 
general, in my life I like to 
stop and observe myself and 
what I did.” 


5.b. Emotional aspects 

References to moods (anxiety, 
fatigue, difficulty) and expectations 

“This role has generated 
anxiety in me; it was a hard 
work.” 



